Data codec method and apparatus

ABSTRACT

A data encoding method includes identifying within an image at least a foveal zone and a peripheral zone, the foveal zone of the image being estimated to coincide with the fovea of a user&#39;s eye when viewed by the user, encoding for transmission the different zones of the image at different qualities, the encoding quality for the foveal zone being higher than for the peripheral zone, and transmitting the encoded image to a remote viewing device; meanwhile, a data decoding method includes receiving an encoded image, different portions of the encoded image having been encoded at different qualities according to whether they are in at least a foveal zone and a peripheral zone, the foveal zone of the image being estimated to coincide with the fovea of a user&#39;s eye when viewed by the user, and the encoding quality for the foveal zone being higher than for the peripheral zone, decoding the portions of the image according to a respective decoding scheme corresponding to the respective encoding of each portion of the image, and outputting the decoded image for display.

BACKGROUND OF THE INVENTION Field of the invention

The present invention relations to a data codec method and apparatus.

Description of the Prior Art

The “background” description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description which may nototherwise qualify as prior art at the time of filing, are neitherexpressly or impliedly admitted as prior art against the presentinvention.

Head mounted displays ‘HMDs’ for virtual or augmented realityapplications typically comprise two displays (one for each eye)receiving respective images at corresponding viewpoints, and typicallydo so at a high frame rate because responsive image updating can reducea sense of nausea created by vestibular/image disparity.

As a result, the bandwidth of image data from the source to the HMD isparticularly high, and this can impact on component requirements for theHMD, battery life, data connectivity options, and the like.

Accordingly, it would be desirable to send the image data in a mannerthat improved or maintained subjective image quality for the user for acomparatively lower bandwidth and/or computational load.

SUMMARY OF THE INVENTION

Various aspects and features of the present invention are defined in theappended claims and within the text of the accompanying description.

In a first aspect, a data encoding method is provided in accordance withclaim 1.

In another aspect, a data decoding method is provided in accordance withclaim 11.

In another aspect, an entertainment device is provided in accordancewith claim 14.

In another aspect, a display device is provided in accordance with claim14.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 is a schematic diagram of an entertainment device and displaydevice in accordance with embodiments of the description.

FIG. 2 is a schematic diagram of an image rendered using foveatedrendering in accordance with embodiments of the description.

FIG. 3 is a schematic diagram of an image indicating different encodingschemes for different portions of the image in accordance withembodiments of the description.

FIG. 4 is a schematic diagram depicting a bandwidth vs quality curvesfor different parts on an image in accordance with embodiments of thedescription.

FIG. 5 is a flow diagram of a data encoding method in accordance withembodiments of the description.

FIG. 6 is a flow diagram of a data decoding method in accordance withembodiments of the description.

DESCRIPTION OF THE EMBODIMENTS

A data compression method and apparatus are disclosed. In the followingdescription, a number of specific details are presented in order toprovide a thorough understanding of the embodiments of the presentinvention. It will be apparent, however, to a person skilled in the artthat these specific details need not be employed to practice the presentinvention. Conversely, specific details known to the person skilled inthe art are omitted for the purposes of clarity where appropriate.

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts throughout the several views, FIG. 1shows and example of a data compression system 1, comprising anentertainment system 10 and a head mounted display ‘HMD’ 802.

An example of an entertainment system 10 is a computer or console suchas the Sony® PlayStation 5® (PS5).

The entertainment system or device 10 comprises a central processor 20.This may be a single or multi core processor, for example comprisingeight cores as in the PS5. The entertainment system also comprises agraphical processing unit or GPU 30. The GPU can be physically separateto the CPU, or integrated with the CPU as a system on a chip (SoC) as inthe PS5.

The entertainment device also comprises RAM 40, and may either haveseparate RAM for each of the CPU and GPU, or shared RAM as in the PS5.The or each RAM can be physically separate, or integrated as part of anSoC as in the PS5. Further storage is provided by a disk 50, either asan external or internal hard drive, or as an external solid state drive,or an internal solid state drive as in the PS5.

The entertainment device may transmit or receive data via one or moredata ports 60, such as a USB port, Ethernet® port, WiFi® port,Bluetooth® port or similar, as appropriate. It may also optionallyreceive data via an optical drive 70.

Interaction with the system is typically provided using one or morehandheld controllers 80, such as the DualSense® controller in the caseof the PS5.

Audio/visual outputs from the entertainment device are typicallyprovided through one or more A/V ports 90, or through one or more of thewired or wireless data ports 60.

Where components are not integrated, they may be connected asappropriate either by a dedicated data link or via a bus 100.

An example of a device for displaying images output by the entertainmentsystem is a head mounted display ‘HMD’ 802, worn by a user 800.

Typically such an HMD comprises two displays (one for each eye), and mayalso receive images at a high frame rate (for example at 120 or 100 Hz,or at 60 or 50 Hz with local frame interpolation and/or so-calledreprojection).

In any event, the combination of two images and a high frame rate meansthat a data high bandwidth is typically required to transmit the imageinformation from the entertainment device to the HMD.

This impacts the computational and I/O resources of the entertainmentdevice generating the data and encoding/packaging it for transmission,and also the HMD receiving and depackaging/decoding it for display—bothin terms of computational load and also, in the case of a wireless HMD,in terms of battery life. Finally, such a bandwidth imposes requirementson the wired or wireless connection between the entertainment device andthe HMD, or conversely the wired or wireless connection imposes an upperlimit on bandwidth that can impact the resolution, frame rate, or otheraspect of the images that are transmitted.

It would be beneficial to encode and transmit the image data in a mannerthat reduced computational resource in at least one part of the system,and improved or maintained subjective image quality for the user for acomparatively lower bandwith and/or computational load.

Accordingly, embodiments of the description comprise a data compressionscheme based on foveated rendering.

Referring to FIG. 2 , fovated rendering typically creates two or threezones (three are shown in FIG. 2 ), and rendering is performed atdifferent quality levels dependant on the zone. Due to how humans viewthe world the area at the focal point or fovea position is viewed mostclearly, and typically there can be a ‘blended’ or intermediateperiphery with less detail whilst even further out in the periphery areathe image is pixelated or blurred even further.

Hence based on where the assumed or measured (i.e. estimated) positionof where in the image will coincide with the fovea when viewed by theuser, the foveal zone can be centred on this position, with a peripheralregion surrounding it, and optionally one or more intermediate zones inbetween. The size of the foveal zone can be based on the expected areaof the image to be viewed within the user's fovea. For an HMD where theimage size and distance from the user's eye is known, this can beestimated quite accurately. For other viewing scenarios, such as on aTV, the size of the TV and the distance of the user from it may need tobe input, estimated or measured.

The foveal zone may, by default, be slightly larger than the expectedarea of the image to be viewed within the user's fovea (e.g. by 10 or25% as non-limiting examples), to encompass natural variation in userfovea sizes and also to reduce the chance of any transition from thefoveal zone to the next zone being noticeable to the user.

The net effect of this rendering approach is that the VR experience isperceived as being fully clearly rendered by the user.

The foveated rendering can select the foveal area using one or severalschemes; static foveated rendering simply places the foveal zone in thecentre of the image, assuming that this is where the user is looking.Object tracked foveated rendering places the foveal zone coincident witha displayed object of interest, typically tagged as such within thegame/application (for example the location of the user's avatar, or thelocation of a targeting reticule, or a new non player character, orscripted event, or the like), again on the assumption that this is wherethe user is looking. Meanwhile, gaze tracked foveated rendering placesthe foveal zone coincident with a position in the image the calculatedto project on the fovea of the user, based on where they are currentlylooking. This latter approach gives the best subjective experience to auser, but assumes a gaze tracking capability (for example using a camerato track eye direction).

In practice the different in quality between zones may arise from avariety of different approaches that benefit different aspects of thesystem; for example using lower quality textures and meshes in theperipheral and optionally blended zones reduce computational overheadwhen generating the virtual environment. Meanwhile using lowerresolutions (for example by rendering one in two pixels in the blendedzone, and one in four in the peripheral zone, and then cloning orinterpolating for the missing pixels) reduces computational overheadduring the rendering phase.

Both of these may be beneficial for example to reduce battery drain fora portable device, and/or improve frame rate (which is typicallyconsidered prima facie desirable, and for HMDs and virtual realityapplications, a higher frame rate is typically associated with reducedincidence of nausea due to the images being more rapidly responsive tochanges in viewpoint).

The present description appreciates that these differences in qualitybetween zones can also assist with the transmission of the resultingrendered image to the HMD.

Generally, the inherent quality of the coding scheme for each zonevaries according to the zone, which the highest quality in the fovealzone and the lowest in the peripheral zone. Any intermediate zones (e.g.one or more blended zones) have a correspondingly intermediate quality.

The variation in quality provides a corresponding variation inbandwidth, with high quality encoding using more bandwidth than lowerquality encoding, and vice-versa.

Sources of variations in quality include but are not limited to:

Bit depth for colour representation (more bits for higher quality)

Encoded pixel density (e.g. encode more or all pixels at high quality,and fewer e.g. 3/4, 2/4, 1/4 at lower qualities, relying on cloning orinterpolation at the receiver to back-fill)

in other words, varying encoding resolution (optionally in conjunctionwith rendering resolution).

Quantisation (mapping source values to more quantised values for higherquality, and fewer quantised values for lower quality)

Variations in lossy encoding (for example dropping higher frequencyinformation in frequency domain encodings such as JPEG or MPEG types toreduce quality).

Variations in interpolation/sampling processes (for example using moreinterpolation and/or less sampling for lower quality).

Interpolating data between a stereoscopic image pair or sampling fromcomplementary pixels of such an image pair for lower quality encoding.

Generating fewer or no enhancement layers during encoding in asubsequent zone (e.g. using a low quality encoding base layer for lowquality encoding, and then one or more additional enhancement layers forsuccessively better quality encoding).

Altering the frame rate to transmit fewer images, for example relying oninterframe interpolation to maintain a frame rate at the display.

Clearly some approaches may complement approaches already used duringimage generation—hence for example if the blended zone renders 2 in 4pixels, and the peripheral zone renders 1 in 4 pixels, then only therendered pixels may be encoded (optionally using other variations inquality as well). Similarly if lower quality textures have been used inthe blended or peripheral zone, having less detail, then an encodingscheme that can reduce bandwidth by reducing high frequency detail maycomplement this approach.

It will also be appreciated that whilst a single encoding scheme may beused for the whole image, but with different variations in quality indifferent zones based on differences in settings and parameters (e.g.one encoder using different settings), optionally one or more zones maybe encoded using completely different schemes having different qualityand bandwidth properties (e.g. different encoders).

For example, the foveal zone may be encoded using a lossless scheme,whilst the blended and peripheral zones are encoded using one or morelossy schemes.

Hence in general, by using a different type of encoding, and/ordifferent settings within a type of encoding, a different quality andassociated bandwidth of encoded image may be generated for the differentzones of the image, including at least the foveal zone and peripheralzone, and optionally one or more intermediate or ‘blended’ zones.

Referring now to FIG. 3 , an image may be encoded as a series of tiles(denoted by the large grid superposed on the background image of amountain). A tile may correspond to a so-called macro block, or a set ofsuch blocks (e.g. 2×2 or 4×4), or may correspond to encoding blocks ofvarying size (if supported by the relevant encoding scheme), or mayrepresent a region of the image of any size supported by the relevantencoding scheme. Tiles of different sizes may be used, for example withlarge tiles encoding regions wholly within a particular zone and smallertiles encoding regions bounding two zones, so that either less of alower quality zone is encoded using a higher quality scheme, or less ofa higher quality zone is encoded using a lower quality scheme.

The zones themselves may be concentric circles as illustrated in thefigures, or may be shapes that conform to the chosen tiling scheme, ormay follow the contours or edges of regions of interest for example inan object tracked foveated scheme).

Each tile may then be encoded according to an appropriate encodingscheme.

Hence for example tiles in the foveal zone (indicated in FIG. 3 bydiagonal hatching) may be encoded with the highest quality scheme,whilst tiles in the intermediate or blended zone (indicated in FIG. 3 byhorizontal and vertical hatching) may be encoded with an intermediatequality scheme, and the tiles in the peripheral zone (indicated with nohatching) may be encoded with a lower quality scheme.

In this way, the overall bandwidth associated with the encoding of theimage may be reduced, whilst limiting the subjective impact of the mixedquality encoding by maintaining the highest quality in the region of theimage most likely to be looked at by the user.

Optionally, the resulting transmitted image information may be subjectto a bandwidth budget cap, Such that in normal operation the bandwidthfor a given image or stereoscopic image pair should not exceed a certainamount (although this amount may vary from image to image depending onwhat other information is being transmitted within an overall bandwidthcapability).

Accordingly, it may be that for a given series of different qualityencoding schemes, successive images occupy more or less of the availablebandwidth budget (for example, images with less variation in colour, orfewer high-frequency components, may require less bandwidth to encode).

In this case, optionally the quality of encoding may be selectivelyincreased for an image, optionally until a subsequent increase wouldexceed the budget.

This increase may be implemented on a tile by tile basis for exampleaccording to one or more of the following rules.

-   -   i. Modify boundary rules for tiles so that the threshold        occupancy of a higher quality zone within a tile required to        count that tile as being in the higher quality zone is reduced.

Hence for example if the current rule is that more than (as anonlimiting example) 50% of a tile has to be within the foveal zone tobe encoded according to the high quality scheme used for the fovealzone, then this rule may be changed to 25%, 10%, 5% or 1%. This willhave the effect of increasing the size of the foveal zone, as quantisedinto tiles, and hence increase the proportion of the image encoded usingthe highest quality scheme. This will increase the overall bandwidth butimproves image quality in the area most likely to be noticed by theuser. The choice of the initial % threshold may be a designer choice.Whether to successively step through such lower thresholds or simplyswitch to a second, lower threshold (e.g. 10%) again may be a designerchoice.

This approach may be implemented first for the foveal zone, and secondlyfor successive intermediate or blended zones. In this way, the qualityof encoding expands or ripples out through the image as the effectivesize of successive quality zones increases, until a further increasewould exceed the bandwidth budget.

-   -   ii. Modify boundaries to increase the size of one of the foveal        zone and optionally an intermediate/blended zone.

This may be as simple as increasing the radius of a circular zone.

However alternatively it may involve adding regions to currently outsidezone for example corresponding to an object of interest such as atargeting reticule, user avatar, scripted event or non-player character(for example a non-player character currently speaking or aiming aweapon at the user), or the centre of the image (for example, there is atendency for eyes to at least partially revert to a central directionduring a blink, and so the centre of the image may represent a hedgeposition for the possible direction of view when the user opens theireyes after blinking).

Typically this will involve successively adding tiles to the existingzone in order to encompass the object of interest and/or centre of theimage. Alternatively, it involves adding tiles to initially encompassthe object of interest and/or centre of the image, and then successivelyreaching back to the existing zone; this approach may result innon-contiguous zones, but encompasses the object of interest and/orcentre of the image first in case successively adding tiles to link backto the zone exceeds the bandwidth budget.

-   -   iii. Modify encoding quality.

If the encoding quality of the foveal zone can be improved (for exampleusing any of the sources of the variation described herein), thenoptionally this can be done whilst within the bandwidth budget. This canbe achieve either by changing settings for a current encoding scheme orby changing to a better quality encoding scheme, as noted elsewhereherein.

Similarly if encoding quality of the intermediate/blended zone can beimproved, then optionally this can be done whilst within the bandwidthbudget.

It may be that the improvement in subjective image quality per encodedbit is greater for an improvement to the encoding quality of theintermediate zone than for improvement to the encoding quality of thefoveal zone (in other words, if there are diminishing returns on encodedbits as encoding quality approaches maximum), in which case optionallythe encoding quality of the intermediate zone is increased before thatof the foveal zone, for example according to a bandwidth cost/imagequality comparison curve.

Hence for example referring to FIG. 4 , this illustrates bandwidth coston the x-axis and subjective quality on the y-axis, with a dashed curverepresenting the foveal zone offset from a solid curve representing theintermediate zone, to bias the comparison in favour of the foveal zone.It will be appreciated that here the curves are the same on theassumption that the same overall encoding scheme is being used, but maydiffer if different encoding schemes are used between zones. In anyevent, here it can be seen that up until the point ‘x’, the system mayimprove the encoding quality of the foveal zone and obtain goodsubjective improvement, but as the rate of return of quality reduceswith bits for the foveal zone, improving the encoding quality of theintermediate zone becomes the better option.

The bias between the zones may be a designer choice in any such decisionprocess.

Using such techniques, the encoding quality of one or more zones may beincrementally improved until further improvement would exceed thebandwidth budget.

Conversely, if the default arrangement for encoding zones alreadyexceeds a bandwidth budget, then encoding quality can be reduced,typically starting with the peripheral region. Optionally the encodingquality can be reduced only down to a predetermined minimum quality. Ifthe scheme reaches its minimum quality and still exceeds the bandwidthbudget, then the encoding quality of the next zone (e.g. the or a firstintermediate zone) may start reduced, optionally down to a predeterminedminimum quality for that zone, which may be higher than thepredetermined minimum quality for the peripheral region, and so on allthe way up to the foveal zone, until the encoded image meets thebandwidth budget.

It will be appreciated that incrementally adjusting tiles to meet abandwidth budget may take time; to improve the speed of the process,typical changes in bit cost for changes in encoding can bepredetermined; for example if a tile encoded using scheme X currentlycosts P bits, this is informative about the nature of the imageinformation in that tile, and a relatively robust prediction can be madethat encoding using scheme Y will cost Q bits, based on previousempirical measurements encoding images according to the various schemesavailable to the system. Hence a lookup table of different bit costs forencoding schemes (indicative of different input image data for arespective tile), repeated for different encoding schemes, allow for aprediction of the bit cost when changing scheme for the current tilebased on the current cost for the current scheme.

In this way, a modification to the default encoding scheme can bequickly made, and a trial encoding performed; typically this trialencoding will result in an overall bit cost close to the predicted bitcost. If it is slightly over budget, then tiles whose coding has beenimproved can be selectively reverted to the lower quality encodingschemes in a reverse order to the enhancements described above, andconversely if the resulting cost is below budget, then encoding foradditional tiles can be improved. This reduces the number of cycles ofadditional encoding required for the image.

Similarly, it can be generally assumed that the current image isvisually similar to the previously encoded image, and so optionally thechoice of encoding quality used for the previous image may be transposedto the current image. A trial encoding may then be performed, and againtiles may either have their encoding quality reduced or enhanced basedon the above schemes according to whether or not the resulting image isabove or under the bandwidth budget.

Hence using one or more of these schemes, the encoding quality of animage may be improved until further improvement would exceed thebandwidth budget for the image, and this improvement is implementedgenerally in the order of fovea, then intermediate/blended (if present),then peripheral, optionally with changes in this priority depending onwhether there is a diminishing return on subjective quality foradditional encoded bits for a higher quality region than a lower qualityregion such that it would be more beneficial to improve the quality ofthe lower quality region.

When transmitting the image, meta data indicating which regions of theimage correspond. to which zone is also transmitted. This may take theform of a binary mask in the case of a two level foveated encodingscheme (e.g. including just fovea and peripheral regions), or a two-bitmask for three or four level foveated schemes (e.g. comprising a fovearegion, one or two intermediate regions, and a peripheral region), andso on.

If the encoding scheme for each zone is fixed, then it may not benecessary to indicate which encoding scheme has been used in each zone.

If variations within an encoding scheme have are used from image toimage, the meta data describing the variation (e.g. a variation from adefault) may be included. Optionally this information may only beprovided when variations occur; hence where a succession of images(perhaps by virtue of being similar) all prompt the same variation inencoding scheme to meet the bandwidth budget, then this variation may beindicated with the first such image, but not subsequently, with a newvariation (including where this reverts to the default) only beingconveyed with the next image to use it.

If the choice of encoding scheme varies for a given zone from image toimage, then this may be conveyed in a similar manner, together with anyvariation from a default.

To reduce the amount of meta data required to convey information aboutvariations of encoding scheme or choice of encoding scheme, apredetermined set of variations and encoding schemes may be chosen, forexample 16, 32, 64, 128, 256 schemes and variants within them.

The appropriate choice may be conveyed on a per tile basis (for exampleif optionally there is variation within a zone (for example withincremental reductions in quality as a function of distance from thefoveal region optionally within the blended zone or particularly theperipheral zone), or on a per zone basis, in which case it will beappreciated that only two, three, four, or a similarly small number ofchoices need to be conveyed per image, and may be part of the meta dataused to convey the bitmap of tile types.

Alternatively, it may be self-evident from the encoded data for a giventile what encoding scheme has been used, obviating the need for suchmeta data. However, analysing tiles to determine the encoding scheme mayplace compositional burden on the receiving device, and so the meta datadescribed above may simplify the decoding process even if the encodingtype could be determined from the tile.

In addition to image encoding, optionally audio encoding may be subjectto similar selective differences in encoding quality.

For example some sound sources may be encoded lost thusly, whilst othersare subject to loss the compression. Examples may be that dialogue areencoded lost thusly whilst ambient sounds are encoded using a lossyscheme; alternatively or in addition, sounds associated with objectsdepicted in the foveal zone may be encoded using a higher quality schemethan sounds associated with objects in the intermediate zone orperipheral zone. Meanwhile optionally some sounds are not subject tosuch a scheme, such as for example a musical score, which may be encodedusing a predetermined encoding scheme.

It will be appreciated that the bandwidth used by the audio representsan overhead within the overall bandwidth budget, and so variations inbandwidth may result in variations in the effective bandwidth budget forthe images, or vice versa.

The encoded image information may then be transmitted to the receivingdevice, such as an HMD, optionally in conjunction with encoded audioinformation (alternatively this may have a separate channel with its ownbandwidth budget).

Optionally, the encoded image information is transmitted in the order ofmeta data necessary to determine the type of tile encoding as describedelsewhere herein, then tile data for the foveal zone followed by datafor the or each intermediate zone, and finally tile data for theperipheral zone.

The receiving device receives the image data, and determines whatencoding selection was made for each tile, either by assessing theencoded data in the tile, or typically by reference to meta dataindicating what zone a tile is in and meta data indicating the encodingselection for each zone, alternatively meta data indicating the encodingselection for each tile.

The receiving device may then decode each tile according to the decodingscheme corresponding to the encoding selection for that tile.

Optionally the receiving device may decode tiles on a per zone basis,decoding tiles within the foveal zone first, then tiles within the oreach intermediate zone, and finally decoding tiles within the peripheralzone.

Optionally decoded tiles may be displayed as they are decoded, replacingcurrently displayed tiles (or similarly zones may be displayed as theyare decoded). This results in the foveated zone being updated in thedisplay as quickly as possible. In conjunction with encoding the imageinformation with the foveal zone tiles being the first tiles whose datais transmitted in the image data, this further improves theresponsiveness of the image in the region that is likely to be viewed bythe user, thus again improving subjective quality.

The above description assumes that the source image has been renderedusing foveated rendering, but it will be appreciated that this is notessential to subsequent foveated encoding of the generated image.

Turning now to FIG. 5 , in a summary embodiment of the presentdescription a data encoding method comprises the following steps.

In a first step 510, identifying within an image at least a foveal zoneand a peripheral zone, the foveal zone of the image being estimated tocoincide with the fovea of a user's eye when viewed by the user, asdescribed elsewhere herein.

In a second step 520, encoding for transmission the different zones ofthe image at different qualities, the encoding quality for the fovealzone being higher than for the peripheral zone, as described elsewhereherein.

And in a third step s530, transmitting the encoded image to a remoteviewing device (such as an HMD), as described elsewhere herein.

It will be apparent to a person skilled in the art that variations inthe above method corresponding to operation of the various embodimentsof the apparatus as described and claimed herein are considered withinthe scope of the present invention, including but not limited to that:

In an instance of the summary embodiment, the image further comprises atleast a first intermediate zone between the foveal zone and theperipheral zone, and the encoding quality for the or each intermediatezone is in between the encoding quality for the foveal zone and theencoding quality for the peripheral zone, as described elsewhere herein;

In an instance of the summary embodiment, the encoding quality isreduced between the foveal zone and subsequent zones by one or moreselected from the list consisting of reducing a bit depth for colourrepresentation of the image in a subsequent zone, encoding only a lowerresolution using a subset of image pixels in a subsequent zone, reducingquantisation during encoding in a subsequent zone, reducing highfrequency information encoded during encoding in a subsequent zone,generating fewer or no enhancement layers during encoding in asubsequent zone, and encoding each of a stereoscopic image pair usinginformation from both images during encoding in a subsequent zone, asdescribed elsewhere herein;

In an instance of the summary embodiment, the encoding quality isreduced between the foveal zone and subsequent zones by using adifferent, lower quality encoding scheme, as described elsewhere herein;

In an instance of the summary embodiment, the image is divided intotiles, and the encoding step comprises encoding tiles within thedifferent zones at the different qualities, as described elsewhereherein;

In an instance of the summary embodiment, the encoded image is subjectto a bandwidth limit, and the method comprises the steps of estimatingthe bandwidth cost of currently selected encodings for the image, and ifthe bandwidth cost is less than the bandwidth limit, improving theencoding quality of the image for at least one zone within the bandwidthlimit, as described elsewhere herein;

In this instance, optionally the step of improving the encoding qualityof the image for at least one zone comprises one or more selected fromthe list consisting of modifying tile inclusion rules for image tiles sothat a threshold occupancy of a higher quality zone within a tilerequired to count that tile as being in the higher quality zone isreduced, increasing the size of at least the foveal zone, and increasingan encoding quality of at least the foveal zone, as described elsewhereherein;

In this instance, similarly optionally the step of improving theencoding quality of the image comprises estimating the change inbandwidth cost of candidate improved quality encodings for one or moreportions (e.g. zones or tiles) of the image, prior to encoding saidportions with the improved quality encodings, as described elsewhereherein;

In an instance of the summary embodiment, the encodings used forrespective zones of an immediately preceding image are selected as thecurrent encodings for the current image, as described elsewhere herein;and

In an instance of the summary embodiment, the method comprises the stepof encoding audio of at least one contributing sound with a qualitydependent on which zone the associated sound source is depicted in, asdescribed elsewhere herein.

Turning now to FIG. 6 , in another summary embodiment, a data decodingmethod comprises the following steps.

In a first step 610, receiving an encoded image, different portions(e.g. zones or tiles) of the encoded image having been encoded atdifferent qualities according to whether they are in at least a fovealzone and a peripheral zone, the foveal zone of the image being estimatedto coincide with the fovea of a user's eye when viewed by the user, andthe encoding quality for the foveal zone being higher than for theperipheral zone, as described elsewhere herein.

In a second step 620, decoding the portions of the image according to arespective decoding scheme corresponding to the respective encoding ofeach portion of the image, as described elsewhere herein.

And in a third step 630, outputting the decoded image for display, asdescribed elsewhere herein.

It will be apparent to a person skilled in the art that variations inthe above method corresponding to operation of the various embodimentsof the apparatus as described and claimed herein are considered withinthe scope of the present invention, including but not limited to that:

-   -   the step of receiving an encoded image comprises receiving        metadata indicating which tiles of a tiled image belong in which        zone, as described elsewhere herein; and    -   the step of receiving an encoded image comprises receiving        metadata indicating which encoding scheme was used for which        zone (or which tile), as described elsewhere herein.

It will be appreciated that the above encoding and decoding methods maybe carried out on conventional hardware (such as entertainment system10) suitably adapted as applicable by software instruction or by theinclusion or substitution of dedicated hardware.

Thus the required adaptation to existing parts of a conventionalequivalent device may be implemented in the form of a computer programproduct comprising processor implementable instructions stored on anon-transitory machine-readable medium such as a floppy disk, opticaldisk, hard disk, solid state disk, PROM, RAM, flash memory or anycombination of these or other storage media, or realised in hardware asan ASIC (application specific integrated circuit) or an FPGA (fieldprogrammable gate array) or other configurable circuit suitable to usein adapting the conventional equivalent device. Separately, such acomputer program may be transmitted via data signals on a network suchas an Ethernet, a wireless network, the Internet, or any combination ofthese or other networks.

Accordingly, in a summary embodiment of the present description, anentertainment device 10 (such as the Sony® PlayStation 5®) comprises adata encoder, comprising in turn the following.

A zone identification processor (for example CPU 20) adapted (forexample by suitable software instruction) to identify within an image atleast a foveal zone and a peripheral zone, the foveal zone of the imagebeing estimated to coincide with the fovea of a user's eye when viewedby the user, as described elsewhere herein.

An encoding processor (for example CPU 20) adapted (for example bysuitable software instruction) to encode for transmission the differentzones of the image at different qualities, the encoding quality for thefoveal zone being higher than for the peripheral zone, as describedelsewhere herein.

And, a transmitter (e.g. CPU 20 in conjunction with the data port 60)adapted to transmit the encoded image to a remote viewing device, asdescribed elsewhere herein.

It will be appreciated that instances of this summary embodiment mayimplement the encoding methods and techniques described herein.

Similarly accordingly, in another summary embodiment a display device(such as for example HMD 802) comprises a data decoder comprising inturn the following.

A receiver (not shown) adapted (for example by suitable softwareinstruction) to receive an encoded image, different portions of theencoded image having been encoded at different qualities according towhether they are in at least a foveal zone and a peripheral zone, thefoveal zone of the image being estimated to coincide with the fovea of auser's eye when viewed by the user, and the encoding quality for thefoveal zone being higher than for the peripheral zone, as describedelsewhere herein.

A decoding processor (not shown) adapted (for example by suitablesoftware instruction) to decode the portions of the image according torespective decoding schemes corresponding to respective encoding schemesof each portion of the image, as described elsewhere herein.

And a graphics processor (not shown) adapted (for example by suitablesoftware instruction) to output the decoded image for display.

Again it will be appreciated that instances of this summary embodimentmay implement the decoding methods and techniques described herein.

The foregoing discussion discloses and describes merely exemplaryembodiments of the present invention. As will be understood by thoseskilled in the art, the present invention may be embodied in otherspecific forms without departing from the spirit or essentialcharacteristics thereof. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting of the scopeof the invention, as well as other claims. The disclosure, including anyreadily discernible variants of the teachings herein, defines, in part,the scope of the foregoing claim terminology such that no inventivesubject matter is dedicated to the public.

1. A data encoding method comprising the steps of: identifying within animage at least a foveal zone and a peripheral zone, the foveal zone ofthe image being estimated to coincide with the fovea of a user's eyewhen viewed by the user; encoding for transmission the different zonesof the image at different qualities, the encoding quality for the fovealzone being higher than for the peripheral zone; and transmitting theencoded image to a remote viewing device.
 2. The data encoding method ofclaim 1 in which the image further comprises at least a firstintermediate zone between the foveal zone and the peripheral zone, andthe encoding quality for the or each intermediate zone is in between theencoding quality for the foveal zone and the encoding quality for theperipheral zone.
 3. The data encoding method of claim 1 in which theencoding quality is reduced between the foveal zone and subsequent zonesby one or more of: i. reducing a bit depth for colour representation ofthe image in a subsequent zone; ii. encoding only a lower resolutionusing a subset of image pixels in a subsequent zone; iii. reducingquantisation during encoding in a subsequent zone; iv. reducing highfrequency information encoded during encoding in a subsequent zone; v.generating fewer or no enhancement layers during encoding in asubsequent zone; and vi. encoding each of a stereoscopic image pairusing information from both images during encoding in a subsequent zone.4. The data encoding method of claim 1 in which the encoding quality isreduced between the foveal zone and subsequent zones by using adifferent, lower quality encoding scheme.
 5. The data encoding method ofclaim 1 in which the image is divided into tiles, and the encoding stepcomprises encoding tiles within the different zones at the differentqualities.
 6. The data encoding method of claim 1 in which the encodedimage is subject to a bandwidth limit, the method comprising the stepsof: estimating the bandwidth cost of currently selected encodings forthe image; and if the bandwidth cost is less than the bandwidth limit,improving the encoding quality of the image for at least one zone withinthe bandwidth limit.
 7. The data encoding method of claim 6, in whichthe step of improving the encoding quality of the image for at least onezone comprises one or more of: i. modifying tile inclusion rules forimage tiles so that a threshold occupancy of a higher quality zonewithin a tile required to count that tile as being in the higher qualityzone is reduced; ii. increasing the size of at least the foveal zone;and iii. increasing an encoding quality of at least the foveal zone. 8.The data encoding method of claim 6, in which the step of improving theencoding quality of the image comprises estimating the change inbandwidth cost of candidate improved quality encodings for one or moreportions of the image, prior to encoding said portions with the improvedquality encodings.
 9. The data encoding method of claim 1, in which theencodings used for respective zones of an immediately preceding imageare selected as the current encodings for the current image.
 10. Thedata encoding method of claim 1, comprising the step of: encoding audioof at least one contributing sound with a quality dependent on whichzone the associated sound source is depicted in.
 11. A data decodingmethod, comprising the steps of: receiving an encoded image, differentportions of the encoded image having been encoded at different qualitiesaccording to whether they are in at least a foveal zone and a peripheralzone, the foveal zone of the image being estimated to coincide with thefovea of a user's eye when viewed by the user, and the encoding qualityfor the foveal zone being higher than for the peripheral zone; decodingthe portions of the image according to a respective decoding schemecorresponding to the respective encoding of each portion of the image;and outputting the decoded image for display.
 12. A data decoding methodaccording to claim 11, in which the step of receiving an encoded imagecomprises receiving metadata indicating which tiles of a tiled imagebelong in which zone.
 13. A non-transitory, computer readable storagemedium containing a computer program comprising computer executableinstructions adapted to cause a computer system to perform a dataencoding method, comprising: identifying within an image at least afoveal zone and a peripheral zone, the foveal zone of the image beingestimated to coincide with the fovea of a user's eye when viewed by theuser; encoding for transmission the different zones of the image atdifferent qualities, the encoding quality for the foveal zone beinghigher than for the peripheral zone; and transmitting the encoded imageto a remote viewing device.
 14. An entertainment device comprising: adata encoder comprising in turn: a zone identification processor adaptedto identify within an image at least a foveal zone and a peripheralzone, the foveal zone of the image being estimated to coincide with thefovea of a user's eye when viewed by the user; an encoding processoradapted to encode for transmission the different zones of the image atdifferent qualities, the encoding quality for the foveal zone beinghigher than for the peripheral zone; and a transmitter adapted totransmit the encoded image to a remote viewing device.
 15. A displaydevice comprising: a data decoder comprising in turn: a receiver adaptedto receive an encoded image, different portions of the encoded imagehaving been encoded at different qualities according to whether they arein at least a foveal zone and a peripheral zone, the foveal zone of theimage being estimated to coincide with the fovea of a user's eye whenviewed by the user, and the encoding quality for the foveal zone beinghigher than for the peripheral zone; a decoding processor adapted todecode the portions of the image according to respective decodingschemes corresponding to respective encoding schemes of each portion ofthe image; and a graphics processor adapted to output the decoded imagefor display.